The vestibulo-ocular reflex (herein abbreviated VOR) is a reflex of the human eye whereby head movement causes movement of the eyes in an opposite direction (i.e., to that of the head). As the head moves, the semicircular canals in the ears, which are spatially located in three perpendicular planes, send signals to the brain indicative of the velocity and acceleration of the head in all directions. The brain then sends signals to the muscles of the eye to move in an opposite direction to the direction of head movement. The VOR results in a stabilized image on the retina of the eye as the head moves and allows the eyes to stay aligned and focused on an object even as the head moves.
Reference is now made to FIG. 1A, which is an illustration of the vestibulo-ocular reflex, as is known in the art. In FIG. 1A, a person in position 10, is looking with his eye 20, at an object 12. The person in position 10 has a line-of-sight (herein abbreviated LOS) depicted by arrow 26 (It is noted that in FIGS. 1A-1D, lines-of-sight will be depicted by double head arrows, whereas light rays directed towards the eye of a person will be depicted by single head arrows). As the person moves his head up from position 10, represented by position 16, and down, represented by position 18, due to the VOR, his eyes move in an opposite direction to the movement of his head in order to keep his LOS directed towards object 12. For example, in position 16, where the person has his head up, his eye 22 is looking down, such that his LOS 28 stays aligned on object 12. In position 18, where the person has his head down, his eye 24 is looking up, such that his LOS 30 stays aligned on object 12. Because of the VOR, the person will see a stabilized image 14 of object 13, which is in focus and is not blurred, jittered or vibrated. It is noted that the VOR is a fast human reflex, has a response time of less than 10 milliseconds, and can stabilize images on the retina of the eye for changes in head velocity of up to 500 degrees per second. Since the VOR is a fast reflex, the brain of a person generally expects images perceived by the eyes to be stabilized on the retina, (i.e. as the head moves, images perceived by the eyes do not move). This general expectation can be termed “real world behavior” as images perceived by the eyes of a person which originate from the real world (as opposed to those generated or produced by a video system) are usually stabilized.
Reference is now made to FIG. 1B, which is another illustration of the vestibulo-ocular reflex, as is known in the art. In FIG. 1B, a person in position 50 is looking with his eye 70 at an object 52, along a LOS 76. A projection unit 56, which can be, for example, a heads-up display (herein abbreviated HUD), also projects an image to eye 70. The image may be, for example, a firing crosshair, altitude levels of an aircraft or any other known symbology projected to an operator of a vehicle. The image projected to eye 70 from projection unit 56 is superimposed on the image eye 70 sees along LOS 76. It is noted that projection unit 56 is not physically attached to the person. Projection unit 56 projects the image along an axis 82 towards a mirror 57, which has been treated with a semi-reflective treatment making mirror 57 a combiner, as is known in the art. The semi-reflective treatment enables eye 70 to see an image projected onto mirror 57 as well as images located beyond mirror 57.
When the image impinges on mirror 57, the image is directed towards eye 70 along an axis 84. As the person in position 50 moves his head up, represented by position 66, and down, represented by position 68, due to the VOR, the eyes of that person move in an opposite direction to the movement of his head, in order to keep his LOS directed towards object 52. For example, the person in position 66, who has his head up, has his eye 72 looking down, such that his LOS 78 stays aligned on object 52. And person in position 68, who has his head down, has his eye 74 looking up, such that his LOS 80 stays aligned on object 52. Since the image projected along axis 82 impinges mirror 57 at a particular location, then as the person in positions 66 and 68 looks through mirror 57 towards object 52, in each position he will see the image projected from projection unit 56 along a particular axis, respectively an axis 78 and an axis 80. Due to the VOR, the image from projection unit 56 will also be stabilized on the retina. Therefore, the image 54 seen by the person will be stabilized, such that an object 64 is seen in focus, and images projected from projection unit 56 will also be seen in focus, such as a crosshair 60, a target marker 62 and a level indicator 58.
Reference is now made to FIG. 1C, which is a further illustration of the vestibulo-ocular reflex, as is known in the art. In FIG. 1C, a person in position 100 is looking with his eye 108 at an object 112, along a LOS 110. The person is wearing a helmet 102. Helmet 102 includes a projection unit 104, which is physically coupled with the helmet, and a visor 106, which has been treated with a semi-reflective treatment making visor 106 a combiner, as is known in the art. Projection unit 104 and visor 106 are collectively referred to in the art as a helmet mounted display (herein abbreviated HMD). Projection unit 104 can project an image to the eye along an optical path 111 towards eye 108. The image may be, for example, a firing crosshair, altitude levels of an aircraft or any other known symbology projected to an operator of a vehicle.
As the person in position 100 moves his head up, represented by position 118, and down, represented by position 132, due to the VOR, his eyes move in an opposite direction to the movement of his head in order to keep his LOS directed towards object 112. For example, the person in position 118, who has his head up, has his eye 128 looking down, such that his LOS 130 stays aligned on object 112. And the person in position 132, who has his head down, has his eye 142 looking up, such that his LOS 144 stays aligned on object 112. Since projection unit 104 is coupled with helmet 102, and therefore moves as the person moves his head, the image projected along axis 111 will move with his head and will appear blurred on the retina of eye 108. For example, since the person in position 118 moves his head up, and therefore instinctively turns his eye 128 down to stay aligned on object 112, projection unit 122 projects an image along an optical path 126, which is reflected by a visor 124 and impinges on the upper part of the retina of eye 128. And since the person in position 132 moves his head down, and therefore instinctively turns his eye 142 up to stay aligned on object 112, projection unit 136 projects an image along an optical path 140, which is reflected by a visor 138 and impinges on the lower part of the retina of eye 142.
As a result of the VOR, the image 114 seen by eye 108 will be partially stabilized. Object 116 will be in focus, as the VOR adjusts the LOS of eye 108 such that it always stays aligned on object 112. Nevertheless, since projection unit 104 moves as the person moves his head, a level indicator 146, a crosshair 148 and a target marker 150 will also move with his head since the optical path from the projection unit towards the eye will not coincide with the optical path the eye will look at as it stays aligned on object 112. This will result in level indicator 146, crosshair 148 and target marker 150 appearing blurred on the retina of the eye. It is noted that this was not the case in FIG. 1B, where the optical path the eye will look at, as it stays aligned on object 52, coincides with the optical path along which projection unit 56 provides an image. Another reason why the symbology appears blurred on the retina of the eye is that the person expects real world behavior regarding level indicator 146, crosshair 148 and target marker 150. Since the VOR of the person instinctively moves his eyes, which are looking at object 112, in an opposite direction to the direction of movement of his head, the symbology projected onto visor 106 will not be stabilized by the VOR. In effect, the symbology projected onto visor 106 will move as the head of the person moves from position 100 to positions 118 and 132 respectively. Therefore such symbology will not be perceived as other images are normally perceived in the real world, for example object 112, which results in a perceived symbology which is blurred or jittered. Blurred symbology perceived by the eye of the person can also cause undesirable effects, such as motion sickness and making sighting, targeting and aiming a weapon very difficult
Reference is now made to FIG. 1D, which is another illustration of the vestibulo-ocular reflex, as is known in the art. In FIG. 1D, a person in position 180 is looking with his eye 188 at a projected image of object 192, along a LOS 190. Person 180 is wearing a helmet 182. Helmet 182 includes a projection unit 184, which is physically coupled with the helmet, and a visor 186, which has been treated with a semi-reflective treatment making visor 186 a combiner, as is known in the art. Projection unit 184 can project an image to the eye along an optical path 191 towards eye 188. The image may be, for example, a firing crosshair, altitude levels of an aircraft or any other known symbology projected to an operator of a vehicle. Since the outside scene seen by the person in position 180 is dark, and as such, eye 188 cannot perceive it, projection unit 184 also projects a light intensified image of object 192. The light intensified image can also be a FLIR (forward looking infrared) image, an ICCD (intensified charge coupled device) image, a night vision image and the like.
As the person in position 180 moves his head up, represented by position 198, and down, represented by position 212, since the only image the person perceives is the image projected onto visor 186, the person will move his eyes in accordance with the movements of his head to keep his eyes aligned on the light intensified image of object 192. For example, the person in position 198, who has his head up, has his eye 208 looking up, such that his LOS 210 stays aligned on the projection of object 192 onto his visor 204. The person in position 212, who has his head down, has his eye 222 looking down, such that his LOS 224 stays aligned on the projection of object 192 onto his visor 218. Since projection unit 184 is coupled with helmet 182, and therefore moves as the person moves his head, the image projected along axis 191 will move with his head. Since the projected image moves with the movement of the head of the person, the projected image will not be stabilized on the retina of the eye. As mentioned with reference to FIG. 1C, a person expects real world behavior from images perceived by the eyes, whether they be from the real world or from a video system. In FIG. 1D, the VOR cannot stabilize the image projected onto visor 186 as the image itself is constantly moving in accordance with movements of the head.
As a result, object 196, which is projected from projection unit 184, will be out of focus, blurred and jittered, as the VOR cannot adjust the LOS of eye 188 such that it always stays aligned on a fixed projected image of object 192 which is itself constantly moving. Also, since projection unit 184 moves as the person moves his head, a level indicator 226, a target marker 228 and a crosshair 230 also will be seen as blurred. It is noted that in the imaging situation of FIG. 1D, where an operator of a vehicle “sees” via a projected image onto his eye and not by direct sight, the image projected to the eye of the operator can cause undesirable effects, such as motion sickness and sighting, targeting and aiming a weapon very difficult. In particular, when the scene seen by the operator is dark, the chance of motion sickness in the operator is increased, since the operator expects his VOR to stabilize the projected image. As explained below, his VOR will, in practice, not be able to stabilize the projected image, and therefore, since the image will be blurred, the chance of motion sickness increases.
In the imaging situation of FIG. 1D, since the image projected onto visor 186 does not remain still, the VOR of the person cannot stabilize the image of object 192. Furthermore, in practice, since the person may be in a vehicle, for example, an aircraft, which vibrates and resonates (due to, for example, air gusts passing the aircraft and the engine of the aircraft), then even if he does not move his head, and his eyes stay focused and aligned on visor 186, the image projected thereon will still appear blurred due to the movements and vibrations of the vehicle. In order to stabilize the blurred image, systems and methods have been devised to mimic the image stabilization process of the VOR in the image projected onto a visor, as is illustrated in FIGS. 2, 3A and 3B. In such systems and methods, the movement of the head of the pilot (for example, the azimuth, elevation and rotation of the head, the location of the head in the reference frame of the aircraft, the speed and the acceleration at which the head is moving) as well as the movement of the aircraft, are measured. In accordance with those measurements, the image provided to the pilot is modified and processed such that it is presented to the pilot in a constant and fixed position, as if the image did not move with the movements of the head of the pilot. By constantly updating the image such that it is placed in a fixed position relative to the pilot allows the VOR of the pilot to stabilize the image the pilot sees as his head moves around. In general, in the art, the process of modifying an image and processing it such that it is presented to an operator of a vehicle in a constant, fixed position (even though the head of the operator moves vis-à-vis the surface onto which the image is projected), or at the correct spatial location such that the operator will see a stabilized image, is known as spatial registration.
Reference is now made to FIG. 2, which is a schematic illustration of a prior art airborne vision and imaging system, generally referenced 250. System 250 includes a camera 252, a head sensor 254, an inertial navigation system (herein abbreviated INS) 256, a vehicle motion system 258, a display processor 260 and a projection unit 262. INS 256 is coupled with vehicle motion system 258. Head sensor 254, camera 252 and vehicle motion system 258 are each coupled with display processor 260, which is in turn coupled with projection unit 262. Camera 252 may be, for example, an IR camera, a visible light camera and the like. Camera 252 can also be an image generator which can generate virtual reality images. The image received by camera 252 may be light intensified, infrared images, ICCD images, or otherwise images not normally visible to the human eye. If camera 252 is an image generator, then camera 252 generates virtual reality images, for example, of a virtual world. Head sensor 254 is located on the helmet of the pilot (not shown) and measures the movement (e.g., azimuth, elevation, rotation and location of the helmet in the reference frame of the aircraft, as well as the velocity and acceleration of the helmet) of the head of the pilot in all directions. INS 256 is located on the aircraft (not shown), and measures the movement of the aircraft in all directions. Vehicle motion system 258 monitors the position of the aircraft, the movement of the aircraft in all directions, as well as other systems related to the aircraft, such as general avionics, weapons system, a fuel system and the like. Projection unit 262 is located on the helmet of the pilot, which is equipped with an HMD, and projects an image received from camera 252, and any other data provided by vehicle motion system 258, onto the visor of the helmet.
Camera 252 receives an image of an outside scene. If camera 252 is an image generator, then camera 252 generates an image of an outside scene. The image may be an IR image, an ICCD image, or it may be light intensified. Camera 252 provides the image to display processor 260. Head sensor 254 continuously measures the movement of the head of the pilot and provides these measurements to display processor 260. INS 256 continuously measures the movement of the aircraft and provides these measurements to vehicle motion system 258, which in turn provides them to display processor 260. Vehicle motion system 258 can also provide display processor 260 with other data regarding the aircraft, for example, the altitude of the aircraft, the LOS of the weapons system, the amount of fuel left in the aircraft, and the like.
Display processor 260 receives the image captured by camera 252 and corrects the image for any distortion (for example, pincushion distortion or barrel distortion) it may have due to the optics of the camera. If camera 252 is an image generator, then no correction for distortion needs to be executed. Display processor 260 also receives measurements of the movement of the head of the pilot (which is indicative of his LOS) as well as measurements of the movement of the aircraft. Display processor 260 uses both of these measurements to modify and process the image captured, or generated, camera 252. For example, if the aircraft is in a turn maneuver, and the body of the aircraft forms a 45° angle with respect to the horizon, and the pilot has his head angled at a −25° angle with respect to the horizon, then helmet display processor may scale, shift, rotate and crop the image captured, or generated, from camera 252 such that it is placed at the correct spatial location on the visor of the helmet such that it coincides with a fixed and constant position relative to the pilot. Furthermore, if the pilot lifts his head, then display processor 260 may shift the image captured, or generated, from camera 252 down to keep the image in at the fixed position relative to the pilot. It is furthermore noted that the position of the head of the pilot can be predicted by known head movement prediction algorithms and that these predicted positions can be used to modify and process the image such that it stays in the fixed position relative to the pilot (i.e., the image is spatially registered correctly). By keeping the image projected from camera 252 in a fixed position, the VOR of the pilot can stabilize the image on the retina of the eye as the pilot moves his head.
Display processor 260 can also superimpose aircraft symbology, as well as other data, such as a digital map, over the image captured, or generated, from camera 252. It is noted that in the case of a digital map, the digital map image may be superimposed over the image captured, or generated, by the camera in a picture-in-picture (herein abbreviated PIP) format. It is also noted that the symbology, as well as other data, may need to be processed, like the image captured, or generated, from camera 252 was processed, such that it coincides with the fixed position relative to the pilot when that other data is projected onto the visor of the helmet.
Since the visor of the pilot is curved, and the projection unit may contain particular optical elements that distort the projected image, any image projected onto the visor needs to be correctly distorted such that the distortion in the image caused by the projection unit is corrected for and the projected image is seen properly on the curved visor. After display processor 260 has finished processing the original image captured, or generated, by camera 252, which can include, as mentioned above, scaling, cropping, rotating and shifting the image, as well as superimposing other images on the original image, display processor 260 then distorts the image such that it can be presented to the pilot on his visor without distortions. Display processor 260 then provides the processed, distorted image to projection unit 262 which projects the image to the visor of the pilot.
Reference is now made to FIGS. 3A and 3B. FIG. 3A is a schematic illustration of a prior art method for spatial registration. FIG. 3B is a graphical representation of the method of 3A. In procedure 280, a raw image is received from a camera, or generated by an image processor. The image may be, for example, a light intensified image, an IR image, an ICCD image, or a FLIR image. Due to the optics of the camera that received the image, the image may be distorted. If the raw image is a virtual world image generated by an image processor, then the raw image will not be distorted. With reference to FIG. 3B, an image 320 represents a raw image received from a camera, or generated by an image processor. It is noted that image 320 can be distorted due to the optics of the camera. In procedure 282, aircraft data as well as head sensor data is received. Aircraft data can include measurements of the aircraft heading, direction, velocity and acceleration as well as information such as the altitude of the aircraft, the LOS of the weapons system, the amount of fuel remaining in the aircraft, and the like. Head sensor data can include measurements of the movement of the head of the pilot in all directions (e.g., azimuth, elevation, rotation and location in the reference frame of the aircraft, as well as velocity and acceleration).
In procedure 284, the image received from the camera is corrected for distortions. Since the image received from the camera will be processed and manipulated, distortions in the image must be initially corrected before the image is processed, otherwise each manipulation to the image will further distort the original image and the final image will be even further distorted. If the image was generated by an image processor, then no distortion correction needs to be executed. With reference to FIG. 3B, an image 322 represents the distortion-corrected image of raw image 320. In procedure 286, symbology representing information regarding the aircraft is generated. The symbology is generated based on the aircraft data received in procedure 282. With reference to FIG. 3B, an image 324 represents the symbology generated in accordance with the aircraft data received. In procedure 288, a digital map, showing the location of the aircraft, is generated in accordance with the aircraft data received in procedure 282. The digital map will be displayed to the pilot in a PIP format. With reference to FIG. 3B, an image 326 represents a digital map generated in accordance with the aircraft data received.
In procedure 290, the distortion-corrected image of procedure 284 is processed in accordance with the aircraft data and head sensor data such that it will be spatially registered correctly. For example, the processing may include scaling, shifting, cropping and rotating the image, all in accordance with the heading of the aircraft and the head position of the pilot. The processing may also include image processing such as brightness correction, gamma correction and the like. With reference to FIG. 3B, an image 328 represents a scaled, shifted, rotated and cropped version of image 322. Image 328 has been processed in accordance with the aircraft data and the head sensor data to keep the projected image in a fixed position relative to the pilot. In procedure 292, the symbology generated in procedure 286 is processed in accordance with the aircraft data and head sensor data such that it will be spatially registered correctly. With reference to FIG. 3B, an image 330 represents a scaled, shifted, rotated and cropped version of image 324. In procedure 294, the digital map generated in procedure 288 is processed in accordance with the aircraft data and head sensor data such that it will be spatially registered correctly. With reference to FIG. 3B, an image 332 represents a scaled, shifted, rotated and cropped version of image 326. In general, it is noted that the procedures of processing the distortion-corrected camera image, the symbology and the digital map can be executed in parallel.
In procedure 296, the camera image, the symbology and the digital map are fused into a single image. The fusion can be executed by superimposing the symbology over the camera image and inserting the digital map over a portion of the camera image in a PIP format. With reference to FIG. 3B, an image 334 represents a fused version of image 328, image 330 and image 332. In procedure 298, the fused image is stored and recorded. With reference to FIG. 3B, an image 336 represents a stored and recorded version of image 334. It is noted that image 334 has been cropped to form image 336. In procedure 300, the fused image of procedure 296 is processed such that it can be projected to each eye of the pilot and stored in memory. The processing can include properly distorting the fused image, for each eye, such that it will be correctly displayed on the visor of the helmet, which is a curved surface having an inherent distortion. The processing can also include correcting any distortion due to the optics of an image projector which projects the image on the visor of the helmet. With reference to FIG. 3B, an image 338 represents a stored right-eye distorted version of image 334, and an image 340 represents a stored left-eye distorted version of image 334. Each of images 338 and 340 has been distorted in accordance with the inherent distortion of the visor of the helmet, and the image projector, such that the image will appear undistorted when projected onto the visor. In procedure 302, the stored processed fused image of each eye is respectfully projected to each eye of the pilot via the visor of the helmet of the pilot. With reference to FIG. 3B, it is noted that images 338 and 340 are first stored in memory before they are projected to the eyes of the pilot.
In general, in airborne imaging and vision systems, the refresh rate of the projection unit is approximately 60 Hz. Therefore, it takes approximately 16.6 milliseconds for the projection unit to generate an entire image to be projected to a pilot. Furthermore, according to the system and method described above, with reference to FIGS. 2, 3A and 3B, each time an image, whether it be the camera image, the symbology or the digital map, is modified (e.g., cropped, rotated, distortion-corrected and the like), the modified image is stored in memory. Storing the image in memory can be a time costly process, as tens of milliseconds may be needed to store the image, depending on the speed of the memory hardware, the data buses, the image resolution, the update rate of the head sensor data and aircraft data, which can be 30 Hz, for example, and the like. Referring back to FIG. 3B, considering images 320, 324 and 326 are stored in memory, then image 322 is stored in memory, then images 328, 330 and 332 are stored in memory, then image 334 is stored in memory, then images 338 and 340 are stored in memory and finally images 338 and 340 are then projected to the eyes of the pilot, the procedure of spatially registering the image projected to the eyes of the pilot in accordance with the movements of the head of the pilot and the movements of the aircraft can take over 100 milliseconds. At this pace, conventional projection units update the image data projected to the pilot at a typical rate of approximately 10 Hz, a rate significantly lower than the refresh rate of the projection unit, and a rate significantly slower than the VOR of humans. At such rates, the image and the information presented to the pilot may not be current and up-to-date, and the VOR may not be able to properly stabilize the image, since the reflex operates at a rate faster than the update rate of the image.
Systems using the method described in FIGS. 3A and 3B therefore exhibit a significant latency in terms of spatial registration and event registration. Event registration refers to the amount of time required for a camera to capture an event, for example, an explosion, and transmit the explosion graphically to the eye of the pilot via a projection unit. Event registration can also refer to the amount of time required for an image generator to receive data regarding a new object in the image and to generate the new object on the image. Event registration can further refer to the amount of time required to update flight symbols projected to the eyes of the pilot.
U.S. Pat. No. 6,867,753 to Chinthammit et al., entitled “Virtual image registration in augmented field display” is directed to a system for registering a virtual image on a perceived real world background. A tracking light is scanned in the real world environment. Light is detected as it impinges on a pair of detector surfaces of a first detector, at a first time and at a second time, when the first time and second time occur within adjacent scan lines. The time at which a horizontal scan line edge is encountered is derived as occurring half way between the first time and the second time. The horizontal location of the first detector is then determined within a specific scan line which is derived from the scan line edge time. The vertical location of the detector is then determined within a scan frame by measuring the time duration using the beginning of the frame. The location of the detector surfaces is thus determined independently from the temporal resolution of the augmented imaging system at a sub-pixel/sub-line resolution. The augmented image is registered to a 3D real world spatial coordinate system based upon the tracked position and orientation of a user. Prediction algorithms are used to compensate for the overall system latency.
U.S. Pat. No. 5,040,058 to Beamon et al., entitled “Raster graphic helmet mountable display” is directed to a helmet mountable display for supplying a full color image for presentation to a wearer of the helmet. An electronic apparatus, for example, a cathode ray tube, having red, green and blue strips of emitting phosphors, along with an electron gun, for scanning a trace in each strip, provides sweep lines for a full color raster graphic image. An electromechanical apparatus disposes the sweep lines to appear at their proper spatial position on the image. Registration and superposition of the traces are executed by a color information delay apparatus and a focusing optical system, which form a sweep line exhibiting full color characteristics which form a portion of the image.
U.S. Pat. No. 5,880,777 to Savoye et al., entitled “Low-light-level imaging and image processing” is directed to an imaging system for producing a sequence of image frames of an imaged scene at a frame rate R, which is a rate of at least 25 image frames per second. The imaging system includes an analog-to-digital processor for digitizing an amplified pixel signal to produce a digital image signal. Each digital image signal is formatted as a sequence of image frames, with each image frame having a plurality of digital pixel values. The digital pixel values have a dynamic range of values which are represented by a number of digital bits B, with B being greater than 8. The imaging system also includes a digital image processor, for producing an output image frame sequence at frame rate R from the digital pixel values in the sequence of image frames. The output image frame sequence is representative of the imaged scene and is produced with a latency of no more than approximately 1/R, having a dynamic range of image frame pixel values being represented by a number of digital bits D, where D is less than B. The output image frame sequence can be characterized by a noise-limited resolution of at least a minimum number of line pairs per millimeter as a function of the illuminance of the input light impinging on the pixels of a charged-coupled imaging device.